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METHOD AND SYSTEM IN A DIGITAL IMAGE PROCESSING CHAIN FOR AD- 
JUSTING A COLOUR BALANCE, CORRESPONDING EQUIPMENT, AND SOFT- 
WARE MEANS FOR IMPLEMENTING THE METHOD 

The present invention relates to a method in a digital image 
processing chain for adjusting a colour balance, in which 
method 

- the subject is imaged by pixels to form value-sets 
of the colour components, 

component-specific histograms are formed from the 
value-sets of the colour components, 

- cumulative histograms are formed from the compo- 
nent-specific histograms, 

the illumination colour corresponding to at least 
one colour component is defined from the cumula- 
tive histograms, 

- using target colour and defined illumination col- 
our are defined for at least two colour compo- 
nents, a gain factor adjusting the colour balance. 

In addition, the invention also relates to a corresponding 
system, equipment, and software means for implementing the 
method . 

As is known, the human visual system is able to adapt to dif- 
25 ferent illumination colours. Objects in the field of vision 
appear to have unvarying colours within the range of variation 
of the illumination colour. An example of such a situation is 
one in which white objects are recognized as being white, in 
both sunlight and also in brilliant white illumination, de- 
30 spite the more bluish hue of sunlight. Once the illumination 
colour leaves the range of human adaptation, the degree of ad- 
aptation is reduced, but nevertheless some change in the col- 
our is sensed. For instance, white objects then appear to be 
somewhat yellow when seen during sunset. 

35 
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The mechanisms and operation of adaptation in the human visual 
system are still not fully known and understood. Adaptation 
takes place partly in the retina, in which the colour recep- 
tors' sensitivities change uniformly during a specific period, 
5 according to their stimulation. It is also known that other 
parts of the adaptation mechanisms are controlled by the cere- 
bral cortex, in which the processes taking place are extremely 
complex. 



10 Part of the adaptation processes appear to use so-called mem- 
ory colours, when evaluating the correct colour balance. Exam- 
ples of memory colours are colours like white, skin colour, 
grass, the sky, and generally other colours that have a spe- 
cific primariness in specific instances. 

15 

The sensors used in digital cameras are not able to automati- 
cally provide an adaptability similar to that of the human 
visual system (nor even a functionality corresponding to the 
retina) . The sensor determines the amount and colour of the 

20 light being reflected from the imaging object, but in no way 
the visual sensation related to it. Thus, images that have 
been taken, and to which no adjustments have been made, appear 
to have a distorted colour balance. This is because, when a 
person is viewing an objects, their visual system adapts to the 

25 illumination of the environment, instead to the content of the 
image. White objects may appear to be blue or yellow, depend- 
ing on whether the image is taken indoors or outdoors. Natu- 
rally, other colours too may appear, depending on the illumi- 
nation. 

30 

Due to the above among other factors, some kind of colour- 
balance adjustment is made in digital camera equipment and in 
image-processing chains in general. The adjustment is very of- 
ten termed automatic white balance, or simply white balance, 
35 if the adjustment is performed manually. The aim is to adjust 
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the colour balance so that the image will look natural in neu- 
tral viewing conditions. 

The problem concerning achieving a proper colour balance is, 
however, very difficult. The operation of the human visual 
system is extremely complex, and so far it has proved impossi- 
ble to fully analyse and understand it. However, in general, 
the problem can be divided into two parts. These are 1) the 
estimation of the illumination colour while taking an image 
and 2) the definition of the type of correction required by it 
and the calculation and adaptation of an appropriate amount of 
correction to the taken image. 

Automatic systems perform these tasks without requiring any 
actions by the user of the camera equipment. In manually per- 
formed illumination-colour adjustment methods, the illumina- 
tion colour is set by an action of the user of the camera. The 
camera may have predefined settings, such as indoor and out- 
door imaging modes, or colour-temperature adjustment. The user 
may also aim the camera at a known colour, which is typically 
white, and from which the camera then defines a correction, 
based on this selected reference colour. 

A large number of solutions to the problems described above 
25 are disclosed in the prior art, because colour-balance adjust- 
ment is absolutely essential in digital imaging equipment. The 
adjustment can be implemented, either in the camera, or else 
as post-processing of a taken image, in some other device (for 
example, in an image-processing program In a PC) . 
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From the user's point of view, perhaps the most satisfactory 
solution to the problem is to arrange an automatic colour- 
balance adjustment function in the camera, which is imple- 
mented in connection with the taking of the image. In such a 
solution, the user is not required to have any particular 
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knowledge of image processing, or even generally of the con- 
trol of the camera's adjustment buttons. 

As is known, there are at least three basic types of adjust- 
5 ment for the illumination colour, which are based on using il- 
lumination-colour estimation algorithms. 

Grey-world algorithms are one of the first of these algo- 
rithms. These are based on simple emulation of the adaptation 

10 ability of vision. In a simplified form, the average colour of 
the image is calculated in them. This assumes that the colour 
should be neutral, in other words, grey. The colour components 
of the image are then adjusted so that its average colour 
moves towards this grey. The aim is to control the amount of 

15 the adjustment in such a way as to take into account more or 
less the entire adaptability of human vision. Grey-world algo- 
rithms are very widely used as a Joasis for colour-balance ad- 
justment algorithms in digital camera devices. 

20 A second such adjustment algorithm is known as brightest white 
balance. As is known, a pure white surface will reflect the 
entire spectrum of illumination. Tlius, the colour of white ar- 
eas will also tell the illumination colour. When fully 
adapted, the white areas should remain white, so that the il- 

25 lumination colour can be directly compensated. The brightest 
areas of images are usually white surfaces, so that brightness 
can be used to search for white areas . 

Colour saturation and hue can also be taken into account. 
30 These properties can also be used to determine the amount and 
type of colour-balance correction, in such a way that the 
adaptability of vision can be more or less taken into account. 
Due to the dominant importance of the white colour, the entire 
group of colour-balance algorithms are often called white- 
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balance algorithms;, even if they do not even measure the illu- 
mination colour from white, or balance the white colour. 



Algorithms based on a maximum colour can be given as a third 
example. If the image has even one surface that perfectly re- 
flects some part of the illumination spectruiti, this part of 
the spectrum will then be known. If a perfectly reflecting 
surface exists for each part, the entire spectrum can be re- 
constructed. Typically, camera sensors do not measure the 
whole spectrum, but only some of its sub-bands, or colour 
channels. Thus, if the image contains combinations of sur- 
faces, which perfectly reflect the parts of the illumination 
and so affect these colour channels, the effect of the illumi- 
nation on the appearance of the image can be estimated. 

Most digital cameras, though not, however, all, use three col- 
our channels to record the colours of the v^iew. These are 
typically red, green, and blue (RGB) . To simplify slightly, 
this means that, if the image includes surfaces that perfectly 
reflect the illumination colour's red, green, and blue compo- 
nents, the illumination colour can be estimated, from them. 

Typically, the maximum value of each colour component is re- 
lated to perfect reflection. With this assumption, it can be 
concluded that the maximum values of the colour components 
also determine the illumination colour. It should be noted, 
that quite often these values are reflected from the brightest 
white surface. In that case, the use of the method based on 
maximum colours will be close to the brightest-white algo- 
rithms. However, the presence of white is in n.o way essential 
when using the maximum- colour method. The most suitable colour 
spaces for the implementation are also different . Like other 
methods, methods based on maximum colour may adjust the amount 
and type of correction, based on the estimated illumination. 
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In addition, the methods described above oan also be combined. 
Some methods may thus belong to more than one of the above 
groups. The colour balance can be made nnore robust by using 
several methods simultaneously and combining their results 
5 sensibly. 

In camera-device-specific image-processing- chains, the colour- 
balance algorithms must operate in co-oper-ation with the other 
image-processing algorithms. The chain can be implemented in 

10 the actual camera device, or also as post-processing functions 
in some other device. Typical algorithms of the chain concern 
not only colour-balance adjustment, but also colour-filter- 
array interpolation (CFAI) , digital gain control, black--level 
adjustment, noise reduction, vignetting correction, geometri- 

15 cal-distortion correction, image enhancements, such as sharp- 
ening and gamma correction, or some oth.er kind of sensor- 
response correction . 

The implementation of the chain can also be divided into func- 
20 tions to be performed in the camera and functions to be per- 
formed as post-processing. In addition, they can be imple- 
mented as both hardware and software solutions. The implemen- 
tation can be completely separate from the implementations of 
other algorithms, so that they can take as input for analysis 
25 and processing only the image data from a previous algorithm, 
with no other information relating, for example, to the previ- 
ous algorithm. 

On the other hand, the algorithms may al-so co-operate very 
30 closely with each other. A typical example of this is an ar- 
rangement, in which the colour-balance algorithm can control 
gains in the colour components. The implementation of the so- 
lutions can both analog and digital. All dLn all, it would be 
extremely advantageous, if the colour-balance adjustment algo- 
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rithm could be implemented as effectively and naturally in a 
camera-specific image-processing chain. 



The main weakness in solutions according to the prior art is 
5 ineffective or erroneous operation. Their effect varries with 
the algorithm. A general problem with all methods (for exam- 
ple, in shots taken during sunset) is achieving a compromise 
between effectively operating colour-balance adjustment and 
the loss of the colour appearance of the view. In addition to 
10 this, each group of algorithms has its own typical cases of 
errors . 

Grey-world algorithms adjust the colour content of the image 
in such a way that the average effect of the image becomes 

15 neutral. Their main problem is that neutral images are only 
optimal in some special cases. The cases are those in which 
there should be an equal number of all colours after: balanc- 
ing. An example of an opposite case to this is a portrait 
shot, in which the average colour should clearly be moved to- 

20 wards skin colours. This method is also extremely sensitive to 
large areas of colour in the image. These distort thie colour 
balance towards the complementary colours of the subject. Sev- 
eral modifications have been used to increase the robustness 
of these algorithms. For example, the weighted avexages of 

25 hue, saturation, or spatial distribution can be used to reduce 
the effect of highly saturated colours and large areas of col- 
our. Similar problems nevertheless remain, unless the effect 
of the colour balancing is limited. In this case, many images, 
which would require balancing, will remain uncorrected. 

30 

Brightest-white-balance algorithms are extremely sensitive to 
saturation of the pixel values. If the dynamic area of the 
camera sensor is exceeded, the pixel values will be Ij-mited to 
the maximum value. A brightest-white-balance algorithm is ex- 
35 tremely suitable for use with colour spaces that define the 
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brightness signal. However, the sensor typically defines the 
colour from the RGB channels. Saturation will therefore also 
appear in the RGB domain. If all components are saturated, the 
brightest-white-balance method will erroneously estimate the 
5 illumination colour as white and as a result balancing will 
not be performed. If one or two components are saturated, an 
erroneous estimate will be found for some other illumination 
colour. These methods also require a white surface to be found 
in the image. Otherwise, the method will fail to estimate the 
10 illumination colour. 

Methods based on maximum colour are also sensitive to satura- 
tion of the pixel values. Usually, these methods select some 
number of the brightest values, which define the illumination 

15 colour. Therefore slight saturation is permitted. In some 
cases, however, this causes problems. The algorithm also de- 
mands that perfectly reflecting surfaces must be foiand in the 
imaging object for all the colour components. As this condi- 
• tion is not met, for example, in the case of natuare images, 

20 these methods too fail at times . 

Problems have also been known to appear in functions that re- 
late quite substantially to the success of colour-balance ad- 
justment, or at least in their compatibility with it. One ex- 

25 ample of these functions is the need for biassing and vignet- 
ting elimination, caused by inferiority properties in the sen- 
sor. In many cases, it would be practically essential to bias 
the sensor in some way, as otherwise the sensor would produce 
a signal, even though what it detects is completely black. The 

30 need for vignetting elimination also usually arises from the 
sensor's ^inferiority', in which case darkening may appear in 
the edge areas . 
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As a prior art referring to 
mentioned US-publications 
2002/0024609 Al • 



PCT/FI2004/050130 

the patent publications can be 
US-2003/0174216 Al and US- 



5 The present invention is intended to create a new type of 
method and system for adjusting colour balance in a digital 
image processing chain. The characteristic features of the 
method according to the invention are stated in the accompany- 
ing Claim 1 and those of the system in Claim 15. In addition, 
10 the invention also relates to corresponding equipment, the 
characteristic features of which are stated in Claim 20 and to 
software means for implementing the method, the characteristic 
features of which are stated in Claim 21. 



15 In principle, in the method according to the invention, the 
object is imaged initially as pixels to form a value-set of 
the colour components (R, G, B) , from which, according to one 
embodiment, at least all such individual pixel values, or im- 
age areas formed of several pixel values, in whxch one or more 

20 pixel values in even one value-set exceed/meet a selected cri- • 
terion, are filtered out of each value-set, according to the 
method of the invention. Thus, for example, distortions caused' 
by saturated pixels can be eliminated, when estimating the il- 
lumination colour . 

25 

Cumulative histograms of the colour components, from the col- 
our curves of which an illumination colour (Rm, Gm, Bm) 
corresponding to each colour component is defined, are then 
formed from the filtered value-sets. 

30 

In the method according to the invention, the illumination 
colour is defined from the flat area of the colour curve at 
the end of each filtered cumulative histogram- A common point 
hi, in the which the ratio d / Ci+i (or the difference) be- 
35 ti-jeen the consecutive pixel-intensity values meets a criterion 
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set for it, is sought from the flat area for at least one col- 
our component. In a cumulative histogram, the interval of the 
points hi, which is thus the same for each colour component, 
can be set to be quite suitable, because, among other things, 
the saturated pixels are filtered out in a previous stage of 
the method. The illumination colours thus defined are then 
used to form componentwise mappings that adjust the colour 
balance. As such, the illumination color can be compensate by 
using any known method. The invention mainly relates to this 
definition of the illumination colour. In one embodiment, the 
flat area is recognized using two colour components and at 
least one colour component is adjusted. 

The use of the method according to the invention permits the 
robust and effective control and type definition of the col- 
our-balance correction gain- In addition, it achieves good re- 
sults in difficult lighting conditions too. 

The colour-balance adjustment method according to the inven- 
tion can be very effectively implemented in camera-specific 
imaging chains. It can be made to operate naturally in co- 
operation with other algorithms. The degree of complexity of 
the method, which remains reasonable, makes it applicable to 
camera devices, for instance. However, the method can also be 
applied, for example, to post-processing taking place outside 
a camera device. 

According to one embodiment, three sub-criteria are used to 
define the illumination colour in the method according to the 
invention. In them, the pixel (intensity) value relations cor- 
responding to consecutive cumulativity distributions are com- 
pared in a set manner to the criterion values set for them. 
The criterion values can be based, for example, on experrien- 
tial data that have been shown to give good results in coLour- 
balance ad j ustment . 
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According to another embodiment, prior to colouir-balance ad- 
justment according to the invention, pedestal elimination, for 
example, can also be performed on the raw-matrix pixel values. 
5 This will significantly reduce the unnecessary signals pro- 
duced by the sensor. 

Yet other functionalities that can be integrated in the method 
according to the invention include vignetting elimination, 
10 dark-colour correction, and gamma correction. 

The method according to the invention can even be implemented 
in a fully automated manner in a camera device, but also, how- 
ever, as post-processing. In addition, the user' s own discre- 
15 tion can easily be connected to the method. 

The method and system according to the invention can be quite 
easily integrated in both existing image-processing chains and 
camera devices, as well as in those presently being designed. 

20 The method according to the invention scarcely adds so much 
complexity to a camera device that it would cause a noticeable 
detriment to, for instance, the processing power of the equip- 
ment or its memory use. According to one embodiment, even the 
method implemented on a purely software level can be inte- 

25 grated extremely easily in, for example, camera devices. 
Purely hardware-level implementations and combinations of 
hardware/software level implementations are naturally possi- 
ble. The method may be implement either automatic or half- 
automatic . 

30 

Other characteristic features of the method, system, equip- 
ment, and software means according to the invention will be 
apparent from the accompanying Claims while additional advan- 
tages that can be achieved are itemized in the description 
35 portion. 



11 



wo 2005/029411 



PCT/FI2004/050130 



In the following, the method, system, equipment, and software 
means for implementing the method, according to the invention, 
which are not restricted to the embodiments described in the 
5 following, are described in greater detail with reference to 
the accompanying drawings, in which 
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20 



shows a schematic diagram of one example of 
an image-processing chain according to the 
10 invention, 

shows one example of colour-component pixels 
in a sensor, 

and 3b show a comparison between the prior art and 
the method according to the invention, when 
forming componentwise histograms from them, 
and 4c show componentwise cumulative histograms as 
schematic embodiments in conaection with the 
method according to the invention, 
shows a schematic diagram of the principle 
of offset correction, 

shows a schematic diagram of the principle 
of dark-colour correction, 

shows the processing of histograms, accord- 
ing to the method of the invention, 
shows examples of gamma-correction curves as 
examples in principle, and 

shows tabled examples of the criterion val- 
ues used in the colour-balance adjustment 
method according to the inventiion. 

30 

The schematic diagram in Figure 1 shows one example of a digi- 
tal image processing chain 10 according to the invention. Such 
an image processing chain 10 can be arranged, for example, in 
a digital camera device, or in general in a device equipped 
35 with a camera sensor, which can be found nowadays, for in- 
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stancer even in mobile stations. The colour-balance correction 
method according to the invention is very suitable for numer- 
ous different camera-specific imaging chains, the implementa- 
tion of which can naturally differ from that shown in Figure 
5 1. In addition, the chain 10 can naturally also be applied to 
post-processing, for example, implemented with software means 
(e.g., in connection with image-processing software). 

Co-operation between the colour-balance adjustment method and 
10 other functions arranged in the image-processing chain 10 can 
also be quite easily implemented effectively. In this case, 
the method according to the invention is shown in an image- 
processing chain 10, which includes CFAI, vignetting elimina- 
tion VE, sensor pedestal elimination PE, and gamma correc- 
15 tion/estimation GE, GA. Any point at all of the chain 10 can 
naturally also include other signal-processing functions, one 
example of which is colour gamut conv^ersion CGC (not shown) , 
arranged before gamma correction. 

20 The actual colour-balance adjustment method according to the 
invention includes several sub-stages, such as the illumina- 
tion colour estimation stage CBA, and correction stage CBC. 
Figure 1 shows how the colour-balance adjustment stages CBA, 
CBC relate to the other functions in "the imaging chain 10. In 

25 Figure 10, the cross-hatched blocks CBA, CBC are the core com- 
ponents of the method. The hatched blocks PE, GE, GC can oper- 
ate in close co-operation with the method according to the in- 
vention. The unhatched blocks VE and CFAI can be independent, 
but also relate to the colour-balance adjustment. 

30 

Next, the method and system according to the invention are de- 
scribed in an image-processing chain 10 according to the in- 
vention, of a kind that can be, for example, in camera equip- 
ment according to the invention, as an individual example of 
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an application, starting from the functions CBA, CBC relating 
to colour-balance adjuistment • 



The first stage of the actual colour-balance adjustment is the 
5 estimation of the illumination colour Rm, Gm, Bm. In the 
method according to the invention, the estimation is performed 
on the basis of the colour-component histograms of the image. 
Componentwise colour curves are then formed from the main col- 
ours R, G, B- 

10 

The brightness/darkness (intensity) of the pixels are depicted 
on the X-axis of the histograms, so that, for example, the 
right-hand end of the X-axis is the bright end and the left- 
hand end the dark end. Similarly, the Y-axis is used to depict 

15 the number of pixels corresponding to each brightness level. 
The formation of the histogram represents a technique that is, 
as such, known, and is therefore not described at greater 
length in this connection. The histograms of the signal values 
(i.e. intensity values) of each colour component (R, G, B) are 

20 measured separately from each other. 

The componentwise histograms can be gathered either from the 
interpolated signals, or also before the interpolation of the 
signals, which is used in this implementation example. This 

25 means that the colour-component histograms can be defined from 
raw image data, in which each physical pixel of the camera 
sensor represents only one colour component. Though in this 
case the sensor's colour system is described at the RGB level, 
other techniques too (CMYK, YUV, etc.) will be obvious to one 

30 versed in the art. 

A typical implementation for performing such an operation is, 
for example, the well-known Bayer-matrix arrangement, shown in 
Figure 2. More generally, this means the depiction of the sub- 
35 ject by pixels to form value-sets [R] , [G] , [B] of the colour 
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components R, G, B. In the case of the RGB application example 
described in the application, two pixels out of four are green 
while one pixel is red and one is blue. There are more green 
pixels, because green best depicts the luminance and is thus 
important from the point of view of the eye. However, such a 
sensor is also possible, in which, for example, only every 
fifth pixel represents the colour green G. Sensors are typi- 
cally more sensitive to green than to red or blue. 

The histograms are collected by scanning through the pixels of 
the image. The number of incidences of each possible signal 
level is calculated for each colour component R, G, B. Thus, 
for example in the case of an RGB image, three histograms are 
obtained, which are shown in Figure 3a. 



However, a important drawback can be seen in the case shown in 
Figure 3a, which relates generally to the prior art in illumi- 
nation-colour estimation. This problem is caused by the detri- 
mental effect of possible saturated pixels, even a single 
saturated pixel G(s) in an image point 15.1 - 15.3 of each shot 
being sufficient to disturb the other pixels R, G, B of the 
image subject. In the histogram 3a, this means a large number 
of pixels with the highest signal value. This leads to prob- 
lems that were described In greater detail earlier in the ap- 
25 plication. This appears as peaks in the histograms visible in 
each colour curve R, G, B, (in Figure 3a) . 



In the method according to the invention, in the histogram's 
collection stage, not only are these saturated pixels G(s, sur- 
30 prisingly rejected, but so are the corresponding pixels R, G, 
B of the image point 15.1. Figure 3b shows the effect of such 
a procedural stage on the histograms in the method according 
to the invention. This operation removes the high peaks of the 
histograms, which would otherwise cause an error in the esti- 
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mation of the illumination colour appearing even in the prior 
art. 



The filtering of the pixel values from each value set [R] , 
[G] , [B] is carried out before the cumulative histograms are 
formed. The filtering is directed at such pixel values G(g), 
or, in addition to them, to the pixels of such image points 
15.1, in which even a single pixel value of the same image 
point 15.1, in even a single value-set [R] , [G] , [B] of the 
image point exceeds or meets the selected criterion set for 
it. 

Estimation of the illumination colour according to the inven- 
tion is not disturbed by the rejection of these saturated pix- 
els G(s) and of other pixels R, G, B too over the same image 
point 15.1, - on the contrary the colour estimation is made 
more reliable. The limit point defining the saturation, i.e. 
the criterion value, can be set to make the method advanta- 
geously compatible with, for example, the pedestal-elimination 
stage to be described later in greater detail. The advanta- 
geous value depends on the general arrangement of the imple- 
mentation. In example case, it is possible to use the crite- 
rion value that is obtained, if the factor pedestal value/2 is 
deducted from the maximum signal value, in which the pedestal 
value can be, for example, 64. The dark end can also be proc- 
essed similarly - 

Thus, if even one of the colour components R, G, B of the pix- 
els of ttie same image point 15.1 is saturated (G(s))/ then the 
other component values R, G, B of the image point 15.1 are 
also rejected. In the case of a raw image, for which CFAI has 
not been carried out, this means that all the colour compo- 
nents R, G,. B, belonging to the same raw-pixel group 15.1 are 
rejected. This and its effects on the collected histograms are 
illustrated on a schematic level in Figures 3a and 3b. 
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In Figures 3a and 3b^ there are two raw-pixel groups 15.1 - 
15. 3, in one of which the saturation problem in question ap- 
pears in a green pixel G(s). Peaks then form in the histogram 
5 of Figure 3a, the formation of which takes into account all 
image points, i.e. raw-pixel groups 15.1 - 15.3. In Figure 3b, 
the raw-pixel group 15.1, which causes the problem, is re- 
jected, so that the histogram curves R, B of the components 
become substantially less steep. 

10 

In the following stage of the method, the illumination colour 
is estimated from the histograms formed from the filtered col- 
our components. Figures 4a - 4c show the cumulative histograms 
of the colour components R, G, B, which are formed from the 

15 actual histograms and from which the saturated pixels G(s) and 
the±r corresponding image points 15.1 have been filtered out 
as described above. According to one embodiment, the cumula- 
tive sum of pixel percentage proportions, or the number of 
pixels, or some other quantity corresponding to a num- 

20 ber/proportion of pixels can be shown, for example, at prede- 
termined selected intervals on the X-axis in the cumulative 
histograms, in which case the intensity values corresponding 
to them can be shown on the Y-axis. In an example case accord- 
ing to the embodiment, the pixels' cumulative percentage pro- 

25 portions hi, for example, set at predetermined intervals on 
the X-axis, state how great a proportion (or number) of the 
filtered pixels is before the intensity point Ri, Gi, Bi corre- 
sponding to them. 

30 In the methods according to the prior art, the illumination 
colour is estimated from the points of the cumulative histo- 
grams, using a set colour representing a specific percentage 
proportion of pixels with values lower than the reference 
value . 

35 
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In the method according to the invention, the illumination 
colour is defined from an essentially flat area at the bright- 
est end of the filtered cumulative histograiKs (the right-hand 
end of the colour curve in the histograms in Figures 4a - 4c) . 
5 In this flat area, or area that can be considered to be essen- 
tially flat (compared, for example, to the other areas of the 
curve) , the curve of the histogram no longer substantially 
rises according to its set criterion conditions, i.e. the in- 
tensity value of the colour no longer substantially changes 
10 according to the conditions set for it. 



In one embodiment, an essential part is played by the proce- 
dure for searching for the flat areas of the colour curves, 
carried out from the cumulative histograms from which satura- 

15 tion pixels G(s) and the their image points 15.1 have filtered 
out. After the setting of the predefined cumulative pixel- 
proportion points hi, hzf . . . , hn attached in a set manner and 
the intensity values (Ri - Rn, d - Gn, Bi - Bn) corresponding 
to these points have calculated, according to the main princi- 

20 pie of the method of the invention a search is made in the 
flat areas of the colour curves R, G, B for a common point hi, 
defined at set intervals for each colour curve R, G, B, and at 
which the ratio Ci / Ci + i (C = R, G, B) of the pixel-intensity 
values corresponding to the consecutive points hi, hi+i defined 

25 at set division intervals meets the criterion values set for 
it. 



The illumination colour Rm, Gin, Bin is defined by analysing 
the sequential intensity values Ci, Ci+i corresponding to these 

30 points hi, according to the method of the invention. It will 
be observed that, if the filtering stage of the saturation of 
the pixel values is bypassed, the method is largely that ac- 
cording to the prior art, in which a set of points (Ri, Gi, 
Bi) , which depends on the currently selected percentage 

35 threshold, in estimated to be the colour of the illumination. 
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In the method according to the invention, the colour is, how- 
ever, estimated using a more highly developed algorithm, which 
sets specific sub-criteria for the points hi corresponding to 
the illumination colour (Rin, Gm, Bin) • 

5 

The points hi, corresponding to the illumination colour (Riu, 
Gill, Bill) are sought by increasing the value i, until the 
flatness condition set for the colour curve is met essentially 
for each curve R, G, B. The intensity value Ri, Gi, Bi, which 
10 is generally different for each colour component, correspond- 
ing to the point hi (i.e. the percentage proportion corre- 
sponding to the point in question) is set as the illumination 
colour (Rill, Gill, Bill) . 

According to one embodiment, this criterion can also be stated 
as three sub-criteria. In the first stage of these sub- 
criteria, it is possible to search each of the colour- 
component curves R, G, B for such a point hi common to the 
curves R, G, B, in which the ratio rc of the pixel intensity 
values Ci, Ci + i corresponding to the consecutive points hi, 
hi+i set in a particular manner, is less than the first thresh- 
old value ti set for it. In other words, the sub-criterion can 
also be formulated mathematically in such a way that 

C, 

rc = -~ < ti , in which C = (R, G, B) 

As the next sub-criterion stage in the method, an examination 
can be made as to whether the sum of the ratios rR, rc, rs of 
the pixel values meeting the first sub-criterion searched in 
30 the first stage is less than the second threshold value ta set 
for it. If this condition is not met, a return is made to the 
first stage to seek a new trio of colour-components meeting 
the first sub-criterion, all of which will thus be at the same 
percentage-proportion point hi, but the intensity values Ri, 

19 
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Gi, Bi corresponding to this point hi generally differ from 
each other. This second sub-criterion can also be formulated 
mathematically, in such a way that 

^iH ^+1 

If the second sub-criterion is not met in the percentage- 
proportion point hi, the procedure returns to the first sub- 
criterion stage. However, if the second sub-criterion too is 
10 met using the trio of colour components that have been found 
(which are found at the point hi) , the procedure moves to the 
third sub-criterion stage. 

In the third sub-criterion stage, an examination is made as to 
whether the ratio between the maximum pixel-value change and 
the minimum pixel-value change of the relative speeds of the 
changes of the pixel values Ri, Gi, Bi found in the first sub- 
criterion stage is less that the third threshold value ta set 
for it. The condition can also be formulated mathematically, 
in such a way that 

max ((Q ~ C,^, ) / C,^, ), in which CsR\\G\\B) 
mm((C, -C,^i)/(C,^^), inwhichCsR\\G\\B) ^ 

Columns 3 and 4 of table 9 show examples of listings of the 
25 threshold values ti, ta, which have been shown experimentally 
to be valid. The threshold value tx can be set in such a way 
that it is, for example, between 1,04 and 1,125, in which it 
is increased at intervals that have been shown experimentally 
to be valid, starting from a base value of 1,04, corresponding 
30 to the index value 1. The threshold value ta can be set in 
such a way that it is, for example, between 3,07 and 3,24, in 
which is it increase at intervals that have been shown experi- 
mentally to be valid, starting from the value 3,07, corre- 

20 
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spending to the index value 1. The threshold value t3 can be 
set to be, for example, approximately 4. 



The index i, corresponding to the preset percentage-proportion 
5 values can be increased, for example 31 times, at intervals 
shown experimentally to be valid. In table 9, the percentage- 
proportion set hi, arranged to be fixed in the cumulative his- 
tograms, can be set according to column 2. The index i - 1 
corresponds to the greatest intensity value (100 %) set in the 

10 histogram from the colour component in question, in which the 
saturated pixels G(s) and the pixels R, G, B formed at the same 
image point 15.1 as the saturated pixel G(s) are filtered out 
when collecting the histograms. The intensity values diminish 
according to the tabled percentage proportions, i.e., for ex- 

15 ample, at the point hio 99,6 % of the intensity values set in 
the histogram which are smaller than or equal in magnitude to 
the point corresponding to the intensity value in question. 
Once 31 steps have been taken using the percentage-proportion 
intervals described, it is finally quite assumable that a col- 

20 our trio (Rm. Gm, Bin) meeting all the sub-criteria has been 
found for the colour of the illumination. The finding of such 
a colour trio will assist in filtering out using the described 
percentage-proportion division of precisely the saturated pix- 
els G(s) and the image points 15.1 consisting of them, in the 

25 histogram-collection stage. 

If, for some reason, the situation is such that an index point 
i meeting the sub-criteria cannot be found, then in that case, 
the index value i = 32 in question is selected without com- 
30 parisons as the reference point, i.e. as the brightest flat 
area and the intensity values of each colour component R, G, B 
corresponding to the said point h32 are selected as the corre- 
sponding illumination colour. 



21 



wo 2005/029411 PCT/FI2004/050130 

The index values i = 32 - 34, at the end of Table 9, can be 
used, for example, in the selection of the gamma value and in 
checking the correctness of the colours. 

5 The value sets of the threshold values ti, ta, ta are based on 
experimental observations, made by the applicant, of a good 
estimation procedure for illumination colours, there being 
thus no particularly logical explanation of them. 

10 Once still a very schematic level, reference is made by way of 
example particularly to the insets in Figures 4a - 4c, which 
insets show enlargements of the flat area, from which the com- 
mon percentage-proportion point hi of the histograms R, G, B 
is sought. It must be understood, that the histograms in the 

15 insets are shown with very greatly exaggerated steepness while 
in other ways too the descriptors may be quite unreal, due to 
the similarity of their form, but in this case they are only 
intended to demonstrate the basic principle of the method ac- 
cording to the invention. 

20 

It can be seen from the histograms R and G that the flat area 
in them, extending to the extreme right-hand end of the histo- 
gram begins, at least compared to histogram B, already quite 
early. Thus in principle the colour curves R and G would meet 
25 the sub-criteria according to the method of the invention al- 
ready at their extreme right-hand end, or at least closer to 
the extreme end than in the case of colour curve B. 

On the other hand, a percentage-proportion interval hi - h4, 
30 between which the intensity of the colour B changes quite 
sharply, at least compared to the colour components R and G, 
and which is substantially steeper than the still flat area at 
the extreme right-hand end of such a flat area, can be seen in 
the histogram B. This rise in the cumulative histogram B can 
35 be caused by, for example, the illumination conditions pre- 
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vailing in the imaging situation, precisely due to which the 
colour balance of the shot must be corrected, in order to 
eliminate the effect of the illumination colour. 

5 When using the method according to the invention, with refer- 
ence to the insets of Figures 4a - 4c, the percentage- 
proportion point hi meeting the set criteria common to each of 
the colour components is now found at point h4, at which a 
flat area meeting the sub-criteria according to the method of 
10 the invention is found from the histogram of the colour compo- 
nent B. The corresponding illumination colour Rin, Gin, Bm is 
now obtained from the intensity value R4, G4, B4 corresponding 
to point h4, which is generally different for each colour com- 
ponent R, G, B. 



15 
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Using the sub-criteria defined above, a surprisingly robust 
estimate of the illumination colour (Kin, Gm, Bm) can be 
made. In addition, the reliability of the estimate can be 
tested using suitable algorithms and the results obtained from 
them used to decide on the amount of correction required and 
whether possible readjustment is required. 



The amount of correction, in other words, the gain factor Gr, 
Gg, Gb, is used to adjust the colour balance. The amount of 

25 correction is directly related to the set target level Rtgt, 
Gtgt/ Btgtf to which each estimated illumination colour is con- 
verted. The gain factor for each colour R, G, B adjusting the 
colour balance is obtained using the target coloTirs Rtgt, Gtgt, 
Btgt and the illumination colours Rm, Gixi, Bm defined in the 

30 manner described above corresponding to each colour component 
R, G, B, in which 



Gr = Rtgt / Rill 

Gg = Gtgt / Gill, and 
Gb = Btgt / Bill. 
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In full-level correction, the aim is to make the illumination 
white (tgt = white; the value on the corresponding scaling 
then being 1000) . The colour-component signals are then ampli- 
5 fied using the gain factor, in such a way that the colour of 
the illumination becomes white. The target level can be re- 
duced as the number of saturated pixels increases, or as the 
brightness of the original image diminishes. This is because 
it is not wished to set the gain too high as the brightness of 
10 the image diminishes, instead an image taken in the dark 
should be slightly darker than an image taken in light. 

Colour-balance adjustment can be made even more robust by 
suitably processing the defined gain factors Gr, Gg, Gb. For 
15 example, the gain factors Gr, Gg, Gb can be compared to each 
other and can be required to meet the following conditions for 
every possible combination: 



in which C = (R\\G\\B) ja CI = (i? || G || 5, ClssC) 

20 

In that case, the correction factor Gr of the red colour CI = 
R, will have a specific interval compared to the correction 
factor Gg of the green, colour G. This same applies to all the 
colours R, G, B relative to the other colours R, G, B. Accord- 

25 ing to one embodiment, in these conditions, it is possible to 
set, for example, k^ain = 0,66 and ^ = 4/3. The value Gcref de- 
picts the mutual relation of the colour components in standard 
illumination, relative to which the colour-sensitivity prop- 
erty of the sensors gives rise to a need for definition. This 

30 is because the sensors are typically more sensitive to the 
green component G than to the red component R and the blue 
component B. Making an automatic correction will depend on, 
for example, the sensor manufacturer. If automatic correction 
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has been made, then the reference value will typically be 1 
for all the colour components. If correction has not been 
made, then, for example, Goref = 1 can be set for green G, in 
which case red can be correspondingly GRref = 1/6 can be set 
5 for red R and Garef = 2,1 for blue B. 

It should be understood that these numerical values can vary 
to some extent, and are thus in no way bound to those stated 
here. These values are also based on experimental results, 
10 which appeared during the pilot stage of the development of 
the algorithm. The gains Gr, Gg, Gb are varied, until the 
aforementioned condition is met. 

By using these modified gain factors Gr, Gg, Gb, a median value 
15 can also be calculated for each colour component R, G, B. If 
it is noticed that the difference between the median value of 
the selected reference colour component and the median value 
of a another colour component is great, the gains are then ad- 
justed, so that the difference between them is reduced. For 
20 example, the green colour component G can be used as a refer- 
ence colour component . 

Further, as a final restriction to the gain factor Gr, Gg, Gb, 
a condition can be set, that each of them must be between: 

25 

in which C = {R,G,B)' 

Here, the value G,ain i.e. an absolute value, can be set as, for 
example, 1 and the value G„ax as, for example, 16, according to 
30 one embodiment, for example, 10. 

If the aforesaid restriction is not met in the case of any of 
the predefined factors Gr, Gg, Gb, the gain factor or factors 
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in question are modified and the other factors are correspond- 
ingly readjusted in the same relation. 

Once the gain factors Gr, Gg, Gb have received their final 
form, they can next be used to form componentwise mappings for 
each colour component R, G, B, which will adjust the colour 
balance. In these mappings, the sub-index In refers to the 
colour detected by the camera sensor while the sub-index Out 
refers to the colour corrected by the gain factor (Gr, Gg, Gb) , 
i.e. 



Rout Gr * Rxti, 
Gout = Gg * Gin, and 
Bout = Gb * Bin. 

The componentwise original histograms are shown in the upper 
descriptor in Figure 7 while the middle descriptor shows the 
componentwise function mappings of them formed from them using 
the gain factors defined according to the method of the inven- 
tion, in which the illumination color is thus taken into ac- 
count according to the method of the invention. The lowest 
histogram shows componentwise histograms processed according 
to the method of the invention, in which so-called offset re- 
duction, by means of which the contrast of the dark end is in- 
creased, are also taken into account. 

In the method according to the invention, other operations be- 
sides the automatic adjustment CBA, CBC of the colour balance, 
described above, can be applied to the image-processing chain 
system 10, arranged in the camera device itself, or alterna- 
tively also in an peripheral device. 

Figure 5 shows a first example of these operations, which is 
known as so-called pedestal elimination PE, i.e. as an offset 
correction of the pixel values performed on the raw matrix. As 

26 



wo 2005/029411 PCT/FI2004/050130 

can be seen from Figure 1 illustrating the image-processing 
chain, this correction can be performed in the chain 10, on 
the raw image obtained directly from the sensor. 

Offset is a minimum value defined from the colour signal, 
which is obtained if, for instance, a completely dark shot is 
captured. The sensors of digital camera devices can have dif- 
ferent offsets for each colour channel. However, it would be 
important from the point of view of the operation of adjust- 
ment of robust colour balance, for the offset to be removed as 
flawlessly as possible and that for the sensor to be thus lin- 
earized. Removal can be arranged to be permanent and sensor- 
specific, provided the same offsets are systematically in- 
creased in the colour-component signals. Adaptive modes can 
15 also be used. 



10 



Independently of the manner of implementation, in the prior 
art, in the principle of offset correction of pixel values, 
the offset values are used to reduce the pixel values. How- 
ever, in pilot-stage tests, such an operation has been shown 
to reduce the dynamic area of the signal, because the bright- 
est values are then also reduced. 



In the method according to the invention, the offset correc- 
25 tion of the pixel values is performed at least partly line- 
arly. According to one embodiment, this can be carried out, 
for example, by removing the pedestal p by a direct subtrac- 
tion calculation, if the signal level of the pixels is below 
the limit value t set for it. If the signal level of the pix- 
30 els exceeds the set limit value t, the pedestal p is sub- 
tracted, but at the same time the signal is also amplified, so 
that the maximum signal level does not substantially drop. 

Such a pedestal elimination stage PE is particularly advanta- 
35 geous, for example, in the case of the method according to the 

27 
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invention, when the illumination colour is estimated and an 
attempt is made to essentially eliminate the effect of the 
saturated pixel values. In the manner described above, i.e. 
performed partly linearly, a saturated pixel will remain satu- 
rated in the pedestal-elimination PE. 

The performance in principle of the pedestal elimination PE 
carried out according to the invention can be described using 
the following equations, with reference to Figure 5 

y = 0, when x<p 
y-x-p, when p^x<t 
x-t 

y = x-p+ p, when x^t 

in which p = pedestal ja t = threshold value 

The co-ordinate axis analogy is clearly applied in such a way 
that y = Out and x = In. In these equations, it can be that, 
15 for example x„«x = 1023, p = 64, and t = x„ax - (p/2) => 1023 - 
(64/2) = 991. 

According to yet another embodiment, the image processing 
chain 10 can also include so-called edge correction, which is 
also known in the terminology of those versed in the art as 
vignetting elimination VE. Vignetting elimination VE can be 
arranged in the image-processing chain 10, for example, after 
pedestal elimination PE. One advantage of using vignetting 
elimination VE is a reduction in corner darkening in images. 
The darkening of the corners of an image is generally caused 
by the optics and the physical properties of the sensor. The 
vignetting phenomenon is characterized by a diminution in the 
Iviminance value of the image, starting from the centre of the 
image and proceeding radially towards the corners of the im- 
30 age. 
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A vignetting-elimination algorithm VE according to the inven- 
tion can be used to compensate for this drawback^ in such a 
way that it applies an offset that varies spatially in the im- 
age and a vignetting-phenomenon correction factor vf, which 
5 can depend on^ among other things, the location of the pixels. 
The value of the correction factor vf increases as the dis- 
tance r of the vignetting phenomenon from the centre point in- 
creases . 

The centre point (r = 0) can be set to be, for example, in the 
centre of the image. The correction factor vf can be developed 
as a function dependent on sensor-specific parameters and the 
distance. The parameters can be defined, for example, using 
calibration imaging, in which the sensor is used to image a 
white surface with standard illumination and from which a 
suitable base estimation of the parameters is used to define 
parameters such that they can be used to eliminate the vignet- 
ting phenomenon. The distance r can be determined in a way 
that is, as such, known (for example, Pythagoras' s theorem), 
in which, however, the so-called aspect ratio must be taken 
into account as an additional coefficient. Vignetting elimina- 
tion is obtained directly, if each pixel value is multiplied 
by the factor vf corresponding to the distance r corresponding 
to it. In addition, account must also be taken of the fact 
that the correction factor can also be developed separately 
for each colour component Rvf, Gvff Bvf, due to their different 
manners of attenuation. If the colour components are attenu- 
ated separately, then it would be good to correct the error 
arising from this, so that it will not cause an error in the 
actual colour-balance adjustment. 

It is advantageous to arrange the vignetting elimination algo- 
rithm VE in such a way that it comes in the image-processing 
chain 10, for example, after the pedestal elimination PE and 
35 before the colour-balance estimation stage CBA, because in 
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that case there will be no possible dark corners in the image 
to distort the actual colour-balance adjustment CBC. 



Figure 6 shows an example of a further sub-stage that can pos- 
5 sibly be attached to the method according to the invention. As 
the final sub-stage in an image-processing chain according to 
the invention, there can, according to one embodiment, also be 
dark-colour correction. In it, values meeting a specific par- 
ticular threshold set form them are sought from the cumulative 

10 histograms. In the dark-colour correction, the darkest colour 
component, determined on the basis of the threshold value, is 
stretched by a set amount towards black and the other compo- 
nents are also processed in the same proportion. The stretch- 
ing takes place by adding an offset of a set magnitude to the 

15 colour-conversion functions, as in Figure 6. 

According to one embodiment, the dark-end threshold value can 
be defined, for example, in such a way that an intensity point 
correspond to, for example, 1 % of the number of pixels is 

20 sought from the histograms of each of the colour components R, 
G, B. After this, the intensity values corresponding to the 1 
%-point of each colour component are multiplied by a current 
correction coefficient corresponding to each colour component 
and the smallest of the modified intensity values thus ob- 

25 tained is selected, which modified intensity value is further 
multiplied by 0,7 and is then selected as the magnitude of the 
correction. 

The magnitude of the correction defined and selected above is 
30 next divided by the current correction factors Gr, Gq,' Gb, as a 
result of which division offset values OffsetR, OffsetG, Off- 
sets are obtained for each colour component. It should be 
noted that the offset values do not effect the colour, but 
only stretch it to become darker. In order that the colour- 
35 balance adjustment target level can be made to remain as it 
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,was originally intended, newly adjusted correction factors 
Gr' , Gg' f Gb' must still be defined. If there are no dark 
points in the image, the contract can be (slightly) increased. 

The lowest histogram in Figure 7 shows an example depicting a 
histogram obtained by the correction of the colour of the dark 
pixels, in which, when correcting the colour of the dark pix- 
els, an offset defined in the aforementioned manner is added 
to the conversion functions, resulting in the readjustment of 
the amplification of the conversion functions. As a result of 
the correction of the dark end, the conversion functions re- 
ceive the following form 

Rout = Gr' * (Rin - Of fsetR) , 
Gout = Gg' * (Gin - Of f setG) , and 
Bout = Gb' * (Bin - Of fsetB) . 

This process then has the result that the points brighter than 
the definition point are converted to be bright and the points 
darker than it are made darker. The effect of the change on 
the histograms is illustrated in the lowest descriptor of Fig- 
ure 7, which also shows the final componentwise histograms ob- 
tained as a result of the image-processing chain 10 according 
to the invention. The mappings functions formed from the col- 
our components are shown in the middle In/Out descriptor. 

Linear conversion functions can be used in the method accord- 
ing to the invention. Non-linearities can also be added and 
typically are added, if the algorithm used is further combined 
30 with possible gamma correction, as an additional embodiment. 

According to one embodiment, gamma correction GC can be imple- 
mented in such a way that first of all the type of image is 
defined. According to one example of the embodiment, this can 
35 be done by searching for the 25 %, 50 %, and 80 %-points on 
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the X-axis of the cumulative histograms, in which, at the 25 
%-point, for example, 25 % of the pixels forming the image 
have an intensity value that is darker than, or equal to that 
of the point. Criteria for the intensity values, on the basis 
5 of which a decision can be made as to whether the shot cur- 
rently being processed is normal, backlight, or low-contrast, 
are set to correspond to the percentage proportions 25, 50, 
and 80. After definition, a smaller gamma-value is selected in 
proportion to how much the image should be brightened. In the 

10 CEI/IEC organization (Commission Internationale de I'Eclairage 
(Vienna) / (International Electrochemical Commission (Geneva) ) 
has selected the gamma level 0,45 = 1/2,2, in Figure 8, for 
SRGB (standard RGB) images. In the method according to the in- 
vention, the gairana values can vary, for example, from 0,20 to 

15 0,60. On a general, very approximate level, it can be stated 
that, in the gamma correction, the colour balance is adjusted 
in the final stage of the adjustment chain 10 by an increase 
in the order of power in which suitable intensity areas invo- 
lution is made for the intensity values in which order of 

20 power corresponds to the gamma value. A direct power-order in- 
crease is not carried out in the initial areas of the curve, 
to prevent the change in the dark area being too rapid 
(steep) . 

25 The sub-stages of the colour-balance adjustment method accord- 
ing to the invention can, according to one embodiment, be sum- 
marized as follows. 

1. Eliminate the pedestal PE, 

30 2. Elimination vignetting VE, 

3. Collect a histogram by filtering out the saturated 



pixel values G(s) and also the image points 15.1 cor- 
responding to them. 



4. 



Define an estimate for the illumination colours Rm, 
Gill, Bill from the cumulative histograms, using bound- 
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ary conditions ti, ta, ta according to the method of 
the invention, 

5. Define using the estimate of the illumination colour, 
necessary colour-conversion mapping functions and us- 
ing them adjust the developed boundary conditions and 
limitations, 

6. Adjust the darkest value of the colour components R, 
G, B, 

7 . Use the addition and histogram data when deciding on 
the visual optimal value for the gamma correction. 

Stage 3, 4, 5, and 6 form the core components of the method 
according to the invention. It should be noted that, depending 
on the implementations, these stages can be rearranged or com- 
bined in several different ways. 

It must be understood that the above description and the re- 
lated figures are only intended to illustrate the present in- 
vention. The invention is thus in no way restricted to only 
the embodiments disclosed or stated in the Claims, but many 
different variations and adaptations of the invention, which 
are possible within the scope on the inventive idea defined in 
the accompanying Claims, will be obvious to one versed in the 
art . 
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